1. Field of the Invention
The present invention relates to an image processing apparatus, image capture apparatus, and an image processing method, and more particularly, to an image correction technique using image recovery processing.
2. Description of the Related Art
The image quality of an image obtained by capturing an object by an image capture apparatus degrades especially due to the aberrations of an optical imaging system. For example, blur of an image indicates degradation in image quality due to the spherical aberration, coma aberration, curvature of field, astigmatism, and the like of the optical imaging system. By ignoring an influence of diffraction as the characteristic of a light wave, light beams coming from one point of the object converge to one point (focal point) of the same size on an imaging plane by an optical imaging system without aberrations. The focal point, however, actually spreads due to the aberrations of the optical imaging system in addition to the influence of diffraction.
The point spread function (PSF) of the optical imaging system represents an intensity distribution around the focal point, that is, blur of the image due to diffraction and the aberrations of the optical imaging system, which is, therefore, referred to as a blur component. The blur component indicates not blur due to an out-of-focus state but blur caused by the diffraction of light and the aberrations of the optical imaging system even in an in-focus state.
Color fringing, in a color image, due to the axial chromatic aberration, color spherical aberration, and color coma aberration of the optical imaging system may be caused by a variation in degree of blur depending on the wavelength of light. A color deviation in the lateral direction due to the chromatic difference of magnification of an optical system may indicate misregistration or a phase shift due to a variation in imaging magnification depending on the wavelength of light.
An optical transfer function (OTF) obtained by performing a Fourier transform for the point spread function (PSF) serves as frequency component information of the aberrations, and is represented by a complex number. The absolute value of the optical transfer function (OTF), that is the amplitude component, will be referred to as an MTF (Modulation Transfer Function), and the phase component will be referred to as a PTF (Phase Transfer Function). The MTF and PTF are the frequency characteristics of the amplitude and phase components of image degradation due to the aberrations. The phase component PTF is represented as a phase angle by:PTF=tan−1(Im(OTF)/Re(OTF))  (1)where Re(OTF) and Im(OTF) represent the real part and imaginary part of the optical transfer function, respectively.
As described above, since the optical transfer function of the optical imaging system degrades both the amplitude component and phase component of the image, respective points of an object image are blurred asymmetrically with respect to the focal point, similarly to a case in which a coma aberration occurs.
The chromatic difference of magnification indicates a phenomenon in which an imaging position shifts due to a variation in imaging magnification depending on the wavelength of light. An image sensor is generally provided with a RGB color mosaic filter, and each pixel is configured to obtain one of R, G, and B color components. In addition to misregistration of an imaging position between R, G, and B wavelengths, misregistration of an imaging position for each wavelength, that is, spread of an image due to a phase shift occurs within each obtained color component. Although the chromatic difference of magnification does not exactly indicate a color deviation due to a simple parallel shift, the color deviation and chromatic difference of magnification are used as synonyms in this specification unless otherwise specified.
There is known a method, called an image recovery method or image restoration method, of correcting degradation in the amplitude component (MTF) and the phase component (PTF) using information of the optical transfer function of an optical imaging system. Processing for correcting degradation in an image using information of the optical transfer function of an optical imaging system, therefore, will be referred to as image recovery processing hereinafter.
An overview of the image recovery processing will now be described. Let g(x, y) be a degraded image, f(x, y) be an original image, and h(x, y) be a point spread function obtained by performing inverse Fourier transform for the optical transfer function of the optical imaging system. Then,g(x,y)=h(x,y)*f(x,y)  (2)where an operator “*” represents convolution and (x, y) represents coordinates on the image.
Fourier transform is performed for equation (2) to obtain a display format on a two-dimensional frequency plane, resulting in the format of a product for each frequency, as represented by:G(u,v)=H(u,v)·F(u,v)  (3)where H represents an optical transfer function obtained by performing Fourier transform for the point spread function, and (u, v) represents coordinates on the two-dimensional frequency plane, that is, a frequency.
To obtain the original image from the captured degraded image, it is only necessary to divide both sides of equation (3) by H, as represented by:G(u,v)/H(u,v)=F(u,v)  (4)
The original image f(x, y) is obtained as a recovered image by performing inverse Fourier transform for F(u, v) to return to the real plane.
Let R be 1/H in above equation having undergone inverse Fourier transform. It is then possible to obtain the original image by executing convolution processing for the image on the real plane, as indicated by:g(x,y)*R(x,y)=f(x,y)  (5)
R(x, y) will be referred to as an image recovery filter. An image recovery filter to be applied to a two-dimensional image is generally a two-dimensional filter having a tap (cell) corresponding to each pixel of the image. Furthermore, as the number of taps (cells) of the image recovery filter increases, the recovery accuracy generally improves. The actual number of taps is set according to the required image quality, the image processing capability, the aberration characteristics, and the like. Since the image recovery filter is based on the optical transfer function which reflects the aberration characteristics of the optical imaging system, degradation in frequency component and phase component can be corrected with high accuracy. Such an image recovery filter is fundamentally different from a two-dimensional filter like an edge enhancement filter (high-pass filter) having three taps in each of the horizontal and vertical directions.
For example, Japanese Patent No. 3532368 discloses a method of canceling blur of an image in a portion, other than an in-focus range, of an image captured by a fluorescence endoscope for observing an inside of a living body, using a point spread function according to a fluorescence wavelength to be used.
Note that since an actual image includes a noise component, using an image recovery filter created by obtaining the complete reciprocal of the optical transfer function amplifies the noise component, thereby disabling to obtain a high-quality recovered image. The image recovery filter created by obtaining the complete reciprocal of the optical transfer function recovers degradation in amplitude by the optical imaging system by correcting (increasing) the MTF of the optical imaging system so that the MTF becomes 1 for all frequencies. If the amplitude component of the image has been added with the amplitude of noise, the power spectrum of the noise increases as the MTF increases, thereby undesirably amplifying the noise according to the recovery degree (recovery gain) of the MTF.
There is a well known method of suppressing noise in a recovered image using an image recovery filter for suppressing the recovery ratio on the high frequency side of the image according to a strength ratio between an image signal and a noise signal, like a Wiener filter. Degradation of a color fringing component of an image can be corrected by, for example, correcting a blur component so that the amounts of blur for respective color components of the image are uniform. Note that since an optical transfer function changes depending on a captured state such as the zoom position or the diameter of the aperture stop of the optical imaging system, it is possible to implement high-precision image recovery processing using an image recovery filter corresponding to the captured state.
There is a well known technique of improving the image quality by performing image recovery processing using the point spread function of an optical imaging system for a captured image to correct various aberrations.
In an actual capturing operation, however, the captured state of an input image may not optimally coincide with the state of an image recovery filter to be applied.
An example is a captured image of a stereoscopic object. Since an image capture apparatus captures an image by focusing on one plane in an object space by an auto focus function or manual focusing, an object positioned in the focal plane is captured relatively sharply. Another object (including a portion of the same object, which is at a position different from the focal plane) is captured with an amount of blur corresponding to a distance from the focal plane.
If information about an object distance includes only the distance from the focal plane, an optimum image recovery filter for the object distance and the angle of view is selected or generated to be used. Consequently, since the optimum image recovery filter is applied to the in-focus object, it is possible to obtain a desired recovery result for such an object. Since, however, the image recovery filter is not optimum for an out-of-focus object, the recovery effect is obtained to some extent but blur cannot be canceled for such an object.
On the other hand, blur of an object in a photo is used as a method of producing a stereoscopic effect for an object or representing an object of interest. For example, there is a photographic technique in which a telephoto lens having a shallow depth of field is used to focus on a main object, and the background is intentionally blurred. In consideration of the presence of such a photographic technique, the above-described image recovery processing in which an in-focus object is made sharper and an out-of-focus object remains blurred may be appropriate.
If, however, image recovery processing is executed using an image recovery filter which is optimum for an object at an in-focus distance and is not optimum for an object at an out-of-focus distance, coloring may occur in the object at the out-of-focus distance. Coloring herein indicates that a color (false color) which the object does not have appears at the edge portion of the object (out-of-focus object) at the out-of-focus distance of the image after the image recovery processing because the relationship between the degrees of blur of the respective color components of the image before the image recovery processing is different from that after the image recovery processing.
Such coloring may also occur in an operation other than an operation of capturing a stereoscopic object. Coloring essentially occurs when the state of aberrations in capturing an image is different from that of aberrations which are corrected by an image recovery filter to be applied, regardless of whether an object is in focus. This may be caused by, for example, a manufacturing variation of the optical imaging system and a variation in spectrum of a light source in capturing an image.
As a method of reducing such coloring, there is provided a method of correcting the color of an image after image recovery processing based on the color information of an image before the image recovery processing. That is, the method is for reducing coloring due to the image recovery processing by determining a change in color due to the image recovery processing in each pixel of the image.
If the image sensor has a Bayer array color filter, an image obtained by the image sensor is a RAW image in which each pixel has one of R, G, and B color components. Two image recovery processes for such a RAW image are considered. The first method applies color interpolation processing to a RAW image so that each pixel has all of R, G, and B color components, and then, applies an image recovery filter to each of R, G, and B planes. The second method applies an image recovery filter to a RAW image in which each pixel has one color component without application of color interpolation processing.
In the method of applying an image recovery filter to each color plane, the number of pixels to be applied with the image recovery processing and the number of taps of the image recovery filter are large, as compared with the method of applying an image recovery filter to a RAW image, thereby significantly increasing the load of the image recovery processing. If, therefore, a low processing load is required, the method of applying the image recovery processing to a RAW image is used. Even when the image recovery processing is applied to a RAW image, coloring may occur in an output image obtained by applying so-called development processing including color interpolation processing to the RAW image if the state of aberrations in capturing the image is different from that of aberrations which are corrected by an image recovery filter to be applied. Coloring that occurs in the image recovery processing which is executed for a RAW image to improve the image quality of an output image results in a degradation in image quality which cannot be ignored, and needs to be sufficiently suppressed.
Since, however, each pixel of a RAW image has only one color component as described above, there is no color information necessary for measuring a change in color before and after the image recovery processing. It is, therefore, impossible to apply, intact, the above-described method of correcting the color of an image after the image recovery processing based on the color information of an image before the image recovery processing.
The method disclosed in Japanese Patent No. 3532368 attempts to complement the shallow depth of field of the optical imaging system by executing the image recovery processing for an out-of-focus range of a captured image. In the image recovery processing described in Japanese Patent No. 3532368, even though it is possible to improve the sharpness of the out-of-focus range, coloring that occurs in the image recovery processing applied to a RAW image cannot be sufficiently suppressed.